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ABSTRACT 


A  simple  Bayesian  measure  of  system  effectiveness  for 
information  retrieval  systems  is  proposed.  The  measure  com¬ 
bines  the  recall  and  precision  ratios  of  an  information  system 
with  the  utility  structure  of  the  system  user.  Using  the  measure, 
it  is  possible  to  show  that  effective  systems  are  possible  only 
under  a  very  narrow  set  of  conditions.  In  particular,  it  is  shown 
that  using  present  state-of-the-art  indexing,  it  is  not  possible  to 
have  effective  systems  with  file  sizes  much  in  excess  of  100,  000 
documents. 
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SECTION  I 


INTRODUCTION 


The  development  in  the  last  two  decades  of  several  new  technolo¬ 
gies  for  storing  and  retrieving  information  has  resulted  in  a  growing 
interest  in  the  design  and  development  of  large  scale  information 
retrieval  systems.  Following  an  initial  period  of  enthusiasm  about 
the  potential  of  the  new  technologies,  a  more  sobering  attitude  has 
developed  due  to  the  realization  that  many  of  the  existing  and  proposed 
systems  are  not  justified  on  a  cost-effectiveness  basis,  In  this  report 
we  wish  to  present  a  model  for  evaluating  the  effectiveness  of  informa¬ 
tion  retrieval  systems  which  is  based  on  Bayesian  statistical  theory. 

This  model  relates  the  effectiveness  of  a  retrieval  system  to  the  sta¬ 
tistical  characteristics  of  the  retrieval  process  and  the  value  structure 
of  the  requestor.  The  model  can  and  does  identify  the  operational 
conditions  under  which-  using  present  state-of-the-art  techniques,  one 
cannot  expect  *o  obtain  sufficient  system  performance  to  justify  the 
development  of  a  system  An  information  retrieval  system  is  basically 
a  system  composed  of  a  file  of  documents  and  procedures  for  partitioning 
it  under  the  control  of  the  requestor.  The  collection  of  documents  is 
mapped  into  a  suitable  multi-dimensional  space  by  the  indexing  process 
Indexing  results  in  the  association  of  one  or  more  tags  (index  terms)  to 
document  of  the  file.  The  index  space  is  usually  a  Boolean  lattice, 
whose  independent  variables  are  the  index  terms  The  index  space  is 
partitioned  by  defining  a  truth  function  over  the  index  space. 

The  input  to  the  information  retrieval  system  is  a  statement 
describing  the  class  of  concepts  on  which  one  desires  additional  infor¬ 
mation  The  output  of  the  system  is  a  partition  of  the  documents  into 
two  clases:  the  retrieved  and  non- retrieved  documents.  Then  by 
definition  all  the  devices,  operators,  and  processes,  which  are  utilized 
in  transforming  the  input  request  into  the  output,  are  elements  of  the 
information  retrieval  system. 

The  approach  taken  in  this  report  is  to  characterize  the  overall 
error  process  by  two  parameters'  the  recall  and  the  precision  ratio 
(Ref,  4,  5).  These  ratios  give,  respectively,  an  indication  of  the 
average  degree  of  completeness  and  the  average  degree  of  purity  (i  e.  , 
lack  of  irrelevant  material)  of  the  search.  These  two  parameters  are 
then  used  to  express  a  complete  model  of  the  error  process  of  the 


retrieval  system  in  a  form  which  is  compatible  with  the  formal  evalua¬ 
tion  technique  developed  in  References  1  and  2.  No  attempt  is  made  at 
this  time  to  analyze  the  error  process  in  its  components:  indexing  error 
process,  request  coding  error  process,  and  search  error  process. 
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SECTION  II 


A  BAYESIAN  EFFECTIVENESS  MODEL 
FOR  RETRIEVAL  SYSTEMS 


In  References  1  and  2  a  scalar  measure  of  effectiveness  for  infor¬ 
mation  systems  is  derived.  This  derivation  is  also  given  in  Appendix 
Bt  The  effectiveness  measure  represents  the  gain  in  expected  utility 
which  results  when  a  rational  decision  maker  utilizes  the  outputs  of 
the  information  system  The  effectiveness  measure  is  given  by  the 
following  formula: 

e(Q)  =  [UD  Q]*S  -  [Uff]*  (1) 

77 

where 

Q  =  {P(Y  jx^)}  The  Information  System  Model 

k  i 

U  =  [U(A  jX  )}  The  utility  structure  of  the  decision  maker, 

77  =  {P(X  )}  The  prior  distribution. 

=  Diagonal  form  of  77. 

5  =  Vector  of  all  ones. 

[  ]  =  Operation  which  takes  the  largest  component  of  the 

column(s)  inside. 


A  simple  but  interesting  model  for  the  evaluation  of  a  retrieval  system 
using  this  effectiveness  measure  is  obtained  as  follows: 


Let  us  introduce  (Refs,  3,  4,  6,  7)  the  recall  ratio  and  the  precision 
ratio  of  the  retrieval  system.  By  precision  ratio  one  means  the  ratio 
of  relevant  material  labeled  retrievable  to  the  material  labeled  retriev¬ 
able.  Thus,  this  ratio  is  equal  to 

NP  [Item  is  relevant  and  is  labeled  retrievable] 
r 

p  =  — — - - — — - - 

NP  [Item  is  labeled  retrievable] 
r 


P (p,  R) 

P(p) 


p(rIp) 


(2) 
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where 


p  =  item  being  examined  is  labeled  retrievable. 

R  =  item  being  examined  is  relevant 

By  recall  ratio  we  mean  the  ratio  of  relevant  material  which  is  labeled 
retrievable  to  the  total  amount  of  material  which  is  relevant.  Thus, 

NP  [item  is  relevant  and  is  labeled  retrievable] 
r 

r  =  - 

NP  [Item  is  relevant] 
r 

=  =  P<PlR>  <3> 

But  by  Bayes  rule, 

n  =  PJR  I  =  P(plR)P(R)  =  P(pjR)P(R; 

P  ^xup,  p(p,  R)  +  P(p,  R)  P(p | R)P(R)  +  P(p 1  R)P(R) 

and  if  we  let 

q  =  P(R) 
l-q=  P(R) 


we  obtain  from  (2)  and  (3)  that 

P(p|R)=  r-ji-  -i=E-  =r^ 

P(o|R)  =  r 
which  together  with 

P(p|R)  =  l-P(p|R)  =  1  —  r 
P(p|R)  =  l-P(p|R)  =  1-r  j8 


\ 

>  (4) 


/ 
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completely  determine  a  simple  binary  model  of  the  retrieval  system 
to  be: 


Q  = 


r 

1-r 

r 

1  -  r/3 

where 

r  =  Recall  ratio  of  the  information  retrieval  system. 


p  =  Precision  ratio  of  the  information  retrieval  system. 

q  =  Probability  that  an  item  in  the  collection  will  be 
relevant  to  the  specific  query. 


The  information  system  output  is  to  be  used  to  decide  which  of  the 
two  actions  retrieve  or  do  not  retrieve  should  be  taken.  Using  this 
simple  model,  the  retrieval  problem  can  be  considered  a  two- state, 
two-act  decision  problem.  The  two  states ^  are  "the  item  is  relevant 
to  the  inquiry,  M  "the  item  is  not  relevant  to  the  inquiry.  "  Thus  the 
utility  structure  for  such  a  decision  task  is: 


Let  us  briefly  digress  on  the  validity  of  the  two- state  decision  theoretic 
formulation  of  the  retrieval  problem.  The  point,  of  course,  is  whether  a 
two-valued  view  of  relevancy  is  appropriate;  Reference  7  attacks  the  two - 
valued  view  of  relevancy  as  incorrect,  but  the  decision  theoretical  formu¬ 
lation  of  the  problem  shows  that  this  is  an  unnecessarily  extreme  position 
since  the  substitution  of  a  two-valued  relevancy  for  a  multivalued  rele¬ 
vancy  is  tantamount  to  coalescing  the  states  into  two  states  with  the 
consequent  averaging  of  the  corresponding  entries  of  the  utility  structure. 

Thus,  the  adoption  of  a  two-valued  relevancy  will  reduce  the  accuracy  in 
computing  the  effectiveness,  but  the  errors  so  committed  are  going  to  be 
bounded  (and  reasonable,  in  general)  in  light  of  the  properties  of  convex 
combinations.  It  seems  to  us  that  it  is  useless  to  advocate  a  multivalued 
view  of  relevancy  in  view  of  the  difficulty  and  inaccuracy  inherent  in  the 
measurement  of  relevancy  since  one  would  be  achieving,  in  most  cases, only  a 
pseudo  higher  accuracy. 
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Retrieve 
Do  Not  Retrieve 


R 

R 

U  , 

u,  „ 

1 1 

12 

u 

U 

21 

22 

The  utility  structure  above  gives  a  value  to  each  of  the  following 
contingencies: 


Document 

Document 

Is 

Is  Not 

Relevant 

Relevant 

Retrieve  Document 

Hit 

False  Drop 

Do  Not  Retrieve 

Document 

Mis  s 

Correct  Rejection 

Thus  for  example,  t^ie  value  of  a  false  drop  Let  us  now  con¬ 

sider  the  following  utility  structure: 


U  = 


y 

-a 


-1 

0 


(6) 


Such  a  structure  is  of  absolute  generality  since  it  can  be  shown  that  the 
addition  of  a  constant  to  each  entry  of  the  utility  structure  does  not  change 
the  value  of  the  effectiveness  function  given  in  (1);  thus  one  can  always 
zero  one  of  the  entries  of  the  utility  structure.  Also  the  division  of  all  the 
entries  of  the  utility  structure  by  the  same  constant  divides  the  effective¬ 
ness  measure  by  the  same  constant.  ^  Thus  setting  of  one  entry  to  _tl  is 
tantamount  to  the  selection  of  the  unit  of  measure  of  effectiveness  Thus, 
substituting  in  ( 1 )  one  obtains 


€(Q)  = 


y  -i 

-a  0 


q  o 

0  1  —  q 


r  1  -  r 
rj3  l--r/S 


§  + 


y  - 1 

-a  0 


q 

i  — q 

(7) 


5* 


The  effectiveness  function  is  a  piece-wise  linear  function  of  the  utility 
entries. 
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which,  after  a  little  algebra,  (7)  becomes 


r(qy-j3)(l-q) 

qy(l-r)-(l-q)(l-rjS) 

* 

yq-(  i-q) 

€(Q)  = 

-Ctqr 

-aq(  1  -r) 

s+  - 

-aq 

(8) 

The  above  (8)  represents  thus  the  simplest  evaluation  model  for  retrieval 
systems  which  is  consistent  with  a  Bayesian  statistical  viewpoint  as 
discussed  in  Reference  4.  The  above  model  has  the  following  five  dimen¬ 
sions: 

r  =  The  recall  ratio. 

.  .  .  . .  q  1  —  p  . 

p  =  The  precision  ratio,  (m  p  =  — — —  — — ■  ■  - ) 

q  =  The  density  of  relevant  material. 

(X  =  The  loss  ratio  of  misses  vs.  false  drops. 

y  =  The  utility  ratio  of  hits  vs.  false  drops. 

The  above  model  was  coded  as  a  FORTRAN  II  procedure  and  then  compiled 
on  an  SDS  920  computer.  The  version  of  this  program  which  was  actually 
used  in  calculating  the  data  reported  in  this  report  is  listed  in  Appendix  I- 

Using  the  program  of  Appendix  I,  three  computational  experiments 
were  executed.  These  experiments  show  that  the  behavior  of  the  model 
(8)  is  determined  predominantly  by  two  parameters: 

9  —  aq  and  X  =  yq 

In  experiment  A,  which  has  the  design  illustrated  in  Figure  1,  the  param¬ 
eter  0  is  much  greater  than  one,  while  the  parameter  \  spans  the  range 
10-5  to  106.  As  it  can  be  seen  from  the  results,  the  effectiveness  of  the 
ISR  (Information  Storage  and  Retrieval  System)  is  uniformly  zero  within 
the  design  ranges  of  r  <  .95  and  p  <  .  95.  This  result  indicates  that 
unless  one  is  prepared  to  furnish  a  system  with  a  recall  ratio  of  better 
than  95%  and  a  precision  ratio  of  better  than  50%^,  one  is  as  well  off 
with  no  information  retrieval  system  at  all. 


This  figure  is  obtained  from  results  not  included  in  Appendix  I. 
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Figure  1.  Experiment  A. 
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Reference  6,  quoting  Cleverdon,  states  "indications  are  that  infor¬ 
mation  retrieval  systems  are  generally  operating  at  a  recall  ratio  of  70 
to  90  per  cent  with  relevance  (i.  e.  ,  precision  ratio)  in  the  range  of  8  to 
20  per  cent.  11  Thus,  it  seems  that  state-of-the-art  techniques  are  quite 
far  from  allowing,  economically,  a  design  level  of  (>  50%,  >  95%). 


In  experiment  B,  which  has  the  design  illustrated  in  Figure  2,  the 
value  of  0  is  unity  and  X  ranges  from  10~~*  to  10^.  It  is  apparent  from 
the  results  of  this  experiment  that  if  X  >  1?  the  effectiveness  of  the  ISR 
becomes  uniformly  null  for  all  but  the  highest  values  of  the  recall  ratio. 
The  behavior  of  the  effectiveness  function  is  quite  similar  in  this  case 
to  the  previous  experiment.  We  can  thus  conclude  that  if  0  >>  1  OR 
\>>  1,  the  effectiveness  of  all  ISR  designs  except  those  with  extremely 
high  values  of  recall  and  precision  ratios  is  null. 


Experiment  C  has  the  design  illustrated  in  Figure  3.  In  this  experi¬ 
ment  0  =  10”3?  i.  e.  ,  0  <<  1  while  X  ranges  from  10"®  to  10^.  It  is  seen 
that  the  design  divides  into  three  regions:  the  region  X  >:>  1  for  which 
the  effectiveness  is  predominantly  zero;  the  region  X  =  1  for  which  the 
effectiveness  is  predominantly  non-zero;  and  the  region  X  <<  1  for 
which  the  effectiveness  is  again  predominantly  zero.  The  first  region 
is  the  result  of  the  previously  stated  property  that  if  0  >>  1  or  X  >>  1, 
the  effectiveness  will  be  predominantly  null.  The  third  region  is  evidence 
that  if  0  <<  1  and  X  <<:  1>  the  effectiveness  will  be  predominantly  null. 

We  can  thus  conclude  that  the  effectiveness  of  an  ISR  is  null  for  all 
but  extremely  demanding  levels  of  design  if  either 

6  >>  1  or  X  >>  1 


or 


0  <<  1  and  X  <<  1 

where  >>  and  <<  mean  different  by  more  than  one  full  order  of  magni¬ 
tude. 


Design  levels  will  be  indicated  with  the  order  pair  (p,  r). 
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Figure  2.  Experiment  B. 
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Figure  3.  Experiment  C. 
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Limits  on  the  Acceptable  Density  of  Relevant  Material 


Both  the  parameters  0  and  X  contain  the  factor  q  which  is  the  proba¬ 
bility  that  an  item  in  the  collection  is  relevant  to  a  specific  query.  The 
results  obtained  can  be  used  to  establish  lower  and  upper  bounds  for  the 
value  of  q. 

If  one  assumes  that  the  state  of  the  art  of  information  retrieval 
consents  to  obtain  the  levels  of  performance^ 

r  =  .  8 

p  =  .  2 

then  our  result  indicates  that  in  order  to  get  a  non-negligible  effective¬ 
ness,  one  must  have 


0  <  1  and  X  <  1 


as  well  as 


IF  -  or  IF  -  x 


The  first  condition  is  equivalent  to 


1  „  1 
q  <  —  and  q  <  — 
—  a  —  y 


which  in  turn  can  be  expressed  as 


q  < 


where  the  operator  +  takes  the  smallest  component  of  the  vector.  The 
second  condition  is  equivalent  to 


q  > 


l 


-  10a 


or  q  > 


1 


—  1  0  y 


This  assumption  is  extremely  optimistic  since  according  to  the  only 
empirical  evidence  available  to  date  (Ref.  3),  one  cannot  attain  both 
of  these  levels  simultaneously  with  the  present  state  of  the  art. 
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which  is  equivalent  to 


Thus 


If 


q  ' 


10a 

1 

lOy 


1 

+ 

l 

10a 

a 

1 

<  q  < 

1 

10y 

y 

6  = 


a 


y 


then  (9)  can  be  written  as 


1 

106  - 


q 


6 


(9) 


(10) 


Systems  which  fall  above  the  upper  bound  correspond  to  files  which 
are  interrogated  just  as  effectively  (when  compared  to  present  day 
indexing  and  retrieval  techniques)  by  exhaustive  examination  of  all  the 
items  in  the  file,  i.  e„  ,  by  dispensing  with  an  ISR.  Systems  which  fall 
below  the  lower  bound  correspond  to  systems  which  are  never  utilized 
since  the  elimination  of  the  non-relevant  material  which  is  retrieved 
requires  an  effort  that  is  greater  than  the  value  of  the  relevant  material 
retrieved.  In  recent  years,  since  the  computer  age  started,  many  such 
retrieval  Leviathans  have  been  developed  and  the  common  experience 
with  them  agrees  with  what  this  simple  Bayesian  theory  predicts,  namely, 
that  they  are  not  being  used. 

It  is  well  to  point  out  here  the  critical  need  for  research  directed  to 
developing  superior  indexing  and  retrieval  techniques  since  it  is  only 
through  the  availability  of  such  superior  techniqies  that  the  size  "range" 
will  be  broadened. 
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SECTION  III 


CONCLUSIONS 


We  have  shown  that  assuming  very  optimistic  levels  of  performance 
for  the  present  state  of  the  art,  one  has  only  a  very  narrow  range  for  the 
density  of  relevant  material  conducive  to  feasible  system  designs.  The 
density  of  relevant  material  can  be  expressed  as  the  ratio  Nr|N  where  Nr 
is  the  number  of  relevant  items  and  N  the  total  number  of  items  in  the 
collection,  This  density  will  vary  greatly  with  the  individual  question. 

We  can  obtain  an  indication  of  what  are  the  probable  feasible  collection 
sizes  by  fixing  a  value  of  N-^  which  is  1 ‘reasonable.  ”  Here,  of  course, 
we  are  interested  in  order  of  magnitudes.  Clearly,  Nr  =  10^  is  the 
largest  value  that  can  be  considered  reasonable. 

In  the  above  formulations,  the  number  a  represents  the  number  of 
irrelevant  documents  which  a  user  is  willing  to  examine  to  avoid  missing 
a  relevant  document.  On  the  other  hand,  the  number  y  represents  the 
number  of  irrelevant  documents  which  abuser  is  willing  to  examine  to 
find  a  relevant  document.  Thus,  6  =  |^j  represents  the  greatest  num¬ 
ber  of  irrelevant  documents  the  user  is  willing  to  examine  for  one 
positive  outcome.  The  value  of  6  will  depend  on  the  amount  of  work 
that  the  examination  of  one  document  entails.  Thus,  6  can  be  increased 
by  techniques  of  summarization  and  abstracting.  It  seems  reasonable, 
though,  to  assume  that  6  will  seldom  exceed  10^  and,  thus,  Nr  6  will 
seldom  exceed  10^  with  the  result  that  the  size  of  files  which  are  amenable 
to  effective  retrieval  should  at  best  be  in  the  range  1  0^  <  N<  10^. 

This  conclusion  implies  that  collections  of  documents  in  the  multi¬ 
million  range  cannot  be  effectively  interrogated  using  a  single  step 
retrieval  process.  In  other  words,  this  file  has  to  be  subdivided  into 
many  files  each  within  the  range  indicated  by  equation  (  10)  and  the 
appropriate  file  should  be  selected  by  a  preselection  stage. 
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APPENDIX  I 


THE  COMPUTER  PROGRAM  AND 
RESULTS  OF  EXPERIMENTS  A,  B,  C 


s 


1 

C 

BAYESIAN  EFFECTIVENESS  OF  RETRIEVAL  SYSTEM  22  MARCH  65  UOG 

2 

C 

NON  LINEAR  AXIS 

3 

DIMENSION  CC2,2D,B[23, RECALL C5 ] , PREC IS[5],T0P[2],EFF[5,5) 

4 

PAUSE 

5 

9 

READ  1, Q, ALPHA, GAMMA 

6 

1 

FORMAT  C3CE15.5D1 

7 

Bt 1 ]-GAMMA*Q- [ 1 . 0-Q] 

8 

BC21— ALPHA*Q 

9 

TOPB-BC 1 ] 

10 

1 F [ B [ 1 3-BI23  36,7,7 

11 

6 

TOPB  -BC2 ] 

12 

7 

DO  5  1-1,5 

13 

DO  5  J-1,5 

14 

RECALLt 1 1-1. 3-. 025*2**1 

15 

PRECIS(J]-1.0-.025*2**J 

16 

8ETA-[Q/[ 1 . 0-Q] ] *[ [ 1 . 0— PREC 1 S[ J] 3 /PREC IS  t J] 1 

17 

C[ 1, 1 1-RECALL [ 1 ]*[Q*GAMMA-BETA*[ 1. 0-Q] ] 

18 

C[ 1,2]-Q*GAMMA*[1. 0-RECALL (  1  ]]-[1. 0-Q] *[  1.0-RECALL t  l]*5ETA] 

19 

C  [2 , 1  ]  — ALPHA*REC  ALL  Ml  *Q 

20 

C  C2 ,2  3-— ALPHA*[ 1 . 0— RECALL  C 1 ]]*Q 

21 

TOPt  1 ]-C[ 1,1] 

22 

IFECE1, 13  — C [2, 13]2,3,3 

23 

2 

TOPt 1 ]-C [2, 1 ] 

24 

3 

T  OP  1 2 ]-C [1,2] 

25 

IFtCt 1,2] — C [2,2334,5,5 

26 

4 

T OP  12] -C t2,2] 

27 

IF  [SENSE  SWITCH  1]  11,5 

28 

11 

TYPE  10, 1 ,J,C, B, TOP, TOPB 

29 

10 

FORMAT  [$ l-$, l2,5X,$J-$, 1 2//$C$ //4 [ F9. 4 ]//$ B$//2 [F9. 4 3// 

30 

1$T0P$//2 [F9. 4]//$T0PR-$, F9. 4] 

31 

5 

EFF [ 1 , J]-TOP[ 1 ]+T0Pt2 1-TOPB 

32 

TYPE  8, Q, ALPHA, GAMMA, [[EFF[I,J], 1-5, 1,-1],  J-1,5] 

33 

8 

FORMAT [$Q-$, E15. 5, 3X,$ALPHA=$, E15. 5, 3X,$GAMMA-$, E15. 5///// 

34 

^EFFECTIVENESS  TABLE  $///[5  F8.2/]] 

35 

12 

FORMAT  [///////////////////////////////////////////////] 

36 

TYPE  12 

37 

GO  TO  9 

38 

END 

PROGRAM  ALLOCATION 


0  (5006  C 
00046  TOP 
00136  Q 
00146  BETA 


00016  B 
00352  EFF 
00140  ALPHA 


00022  RECALL 
00134  I 
00142  GAMMA 


00034  PRECIS 
00135  J 
00144  TOPB 
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EXPERIMENT  A 


e  = 


10 


2 


q  =  10"1 

Nall  at  all 

Sample 

Points 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

-3 

q  =  10 

Nall  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Null  at  all 

Sample 

Points 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

q  =  10"5 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

q  =  10  ^ 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Null  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

Nall  at  all 

Sample 

Points 

- - - 3 - 5 — - 7 

7=0  7=10  7=10  7=10  7=10 


The  sample  points  are  the  Carthesian  product  of  the  two  following 
sets: 


p  =  .  2,  .6,  ,8, 

r  =  .  2,  .6,  .8, 


•  9,  .95 
.9,  .95 
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EXPERIMENT  B 


0  =  1 


q  =  10 

Mostly 

Not  Null 

Not  Null, 
Only  in  the 
Upper 

Right 

Corner 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

q  =  lO"2 

Not  Null 

Mostly 

Not  Null 

Not  Null, 
Only  in  the 
Upper 

Right 

Corner 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

vD 

II 

i— • 

O 

1 

Not  Null 

Not  Null 

Mostly 

Not  Null 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

-5 

q  =  10 

Not  Null 

Not  Null 

Not  Null 

Mostly 

Not  Null 

Null  at  All 

Sample 

Points 

q  =  10"7 

Not  Null 

Not  Null 

Not  Null 

Not  Null 

Mostly 

Not  Null 

7  —  10  y  =  102  y=  103  y  =  105  y  =  10? 


The  following  pages  contain  the  actual  printout  of  the  results.  The  rows 
of  the  result  matrix  correspond  to  distinct  values  of  p  and  the  columns 
to  distinct  values  of  r.  Both  variables  vary  over  the  set 

.2,  .6,  .8,  .  9,  .95 
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Q 


0.99999E-01  ALPHA’ 


0.1 000 0E  02  GAMMA 


0.10000E  02 


EFFECTIVENESS  TABLE 


-0. 

00 

0. 

10 

“0. 

00 

0. 

09 

-0. 

00 

0. 

08 

-0. 

00 

0. 

06 

-3. 

00 

-0. 

00 

50 

0.70 

0.80 

49 

0.69 

0.79 

48 

0.68 

0.78 

45 

0.64 

0.74 

18 

0.34 

0.42 

0 

0 

0 

0 

0 
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Q-  0.99999E-01  ALPHA-  0.10000E  02  GAMMA-  0.10000E  03 


EFFECTIVENESS  TABLE 


'0.00 

-0.00 

-0.00 

-0.00 

0.34 

'0.00 

-0.00 

-0.00 

-0.00 

0.34 

'0.00 

-0.00 

-0.00 

-0.00 

0.33 

0.00 

-0.00 

-0.00 

-0.00 

0.29 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 
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Q 


0.99999E-01  ALPHA 


0.10000E  02  GAMMA 


0.10000E  04 


EFFECTIVENESS  TABLE 


-0.03 

-0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-3.00 
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Q-  0.99999E-01  ALPHA-  0.10000E  02  GAMMA-  0.10000E  06 


EFFECTIVENESS  TABLE 


-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 
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Q 


0.99999E-01  ALPHA 


0.10000E  02  GAMMA 


0.10020E  08 


EFFECTIVENESS  TABLE 


-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

-0. 

00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

0 

0 

0 

0 

0 
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Q' 


0.99999E-02  ALPHA' 


0.10000E  02 


0.10000E  03 


EFFECTIVENESS  TABLE 


0,11 

0.55 

0.77 

0.88 

0.93 

0.11 

0.55 

0.77 

0.88 

0.93 

0.11 

0.55 

0.77 

0.88 

0.93 

0.11 

0.55 

0.76 

0.87 

0.93 

0.10 

0.53 

0.74 

0.84 

0.90 

GAMMA- 
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Q 


0.100C0E  03 


0.99999E-22  ALPHA*  0.10003E  03 


EFFECTIVENESS  TABLE 


-0.  00 

0.19 

0.59 

0.79 

0.89 

-0.00 

0.19 

0.59 

0.79 

0.89 

-0.  00 

0.19 

0.59 

0.79 

0.89 

-0.00 

0.19 

0.58 

0.78 

0.88 

-0.  00 

0.17 

0.56 

0.75 

0.85 

GAMMA" 
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Q-  0.99999E-02  ALPHA-  0.10000E  03  GAMMA-  0.10000E  04 


EFFECTIVENESS  TABLE 


“0.00 

-0.00 

-0.00 

-0. 

00 

0.44 

-0.00 

-0.00 

0.00 

-0. 

00 

0.44 

-0.00 

-0.00 

0.00 

“0. 

00 

0.44 

-0.00 

-0.  00 

-0.00 

-0. 

00 

0.43 

-0.  00 

-0.00 

-0.00 

-0. 

00 

0.40 
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Q-  0.99999E-02  ALPHA-  0.1B000E  33  GAMMA-  0.10009E  06 


EFFECTIVENESS  TABLE 


-0.33 

-3.00 

0.00 

—  0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.09 

-0.00 

-3.09 

-0.00 

-0.00 

-0.00 

-0.00 

-3.00 

-0.  00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.03 

-0.00 
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Q-  0. 99999E-02  ALPHA-  0.10000E  03  GAMMA-  0.10000E  08 


EFFECTIVENESS  TABLE 


-0.00 

-0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 
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0.100C3E  32 


Q-  0.99999E-03  ALPHA-  0.10083E  04 


EFFECTIVENESS  TABLE 


0.19 

0.59 

0.80 

0.90 

0.95 

3.19 

0.59 

0.80 

C.90 

0.95 

0.19 

0.59 

0.80 

0.90 

0.95 

0.19 

0.59 

0.80 

0.90 

0.95 

0.19 

0.59 

0.79 

0.89 

0.94 

GAMMA- 
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Q*  0.99999E-'j3  ALPHA*  0.10030E  34  GAMMA*  0.10330E  03 


EFFECTIVENESS  TABLE 


3.12 

0.56 

0.78 

0.39 

0.94 

3.12 

0.56 

0.78 

0.89 

0.94 

0.12 

3.56 

0.78 

C.89 

3.94 

0.12 

0.56 

3.78 

0.89 

0.94 

3.12 

0.56 

0.78 

3.89 

0.94 
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Q 


0.99999E-03  ALPHA' 


0.10000E  04  GAMMA' 


0.10000E  04 


EFFECTIVENESS  TABLE 


-0.00 

0.20 

-0.  00 

0.20 

-0.00 

0.20 

-0.00 

0.20 

-0.00 

0.20 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

0 

0 

0 

0 

0 


-  31 


Q=  0.99999E-03  ALPHA-  0.10000E  04  GAMMA-  0.10000E  06 


EFFECTIVENESS  TABLE 


-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

”0.00 

-0.00 

-0.  00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.30 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

•0#  0  0 

-0.00 

-0.00 
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Q 


0.99999E-03  ALPHA 


0.10000E  04  GAMMA 


0.10000E  08 


EFFECTIVENESS  TABLE 


-0.  00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.  00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

00 

-0.00 

-0.00 

0 

0 

0 

0, 

0. 
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Q-  0.99999E-05  ALPHA-  0.10000E  06  GAMMA-  0.10000E  02 


EFFECTIVENESS  TABLE 


0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

-  34  - 


0. 10000E  03 


Q-  0.99999E-05  ALPHA-  0.10000E  06 


EFFECTIVENESS  TABLE 


0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

GAMMA- 
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Q' 


0.10030E  04 


0.99999E-05  ALPHA-  0.10000E  06 


EFFECTIVENESS  TABLE 


0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

GAMMA- 
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Q 


0.99999E-05  ALPHA 


0.  10000E  06  GAMMA' 


0.10000E  06 


EFFECTIVENESS  TABLE 


-0.00 

0.20 

-0.00 

0.20 

-0.00 

0.20 

-0,00 

0.20 

-0.00 

0.20 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

60 

0.80 

0.90 

0 

0 

0 

0 

0 
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Q-  0. 99999E-05  ALPHA-  0.10000E  06  GAMMA-  0.10000E  08 


EFFECTIVENESS  TA8LE 


-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

0.00 

0.00 

-0.00 

-0.00 

-0.00 

0.00 

—  0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-0.00 

-  38  - 


Q 


0. 99999E-07  ALPHA 


0. 10000E  08  GAMMA' 


0.10000E  02 


EFFECTIVENESS  TABLE 


0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 
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Q-  0. 99999E-07  ALPHA*  0.10000E  08  GAMMA*  0.10000E  03 


EFFECTIVENESS  TABLE 


0.20 

9.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0;95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 
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Q-  0.99999E-07  ALPHA-  0.10000E  08  GAMMA-  0.10000E  04 


EFFECTIVENESS  TABLE 


0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 

0.20 

0.60 

0.80 

0.90 

0.95 
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Q-  0.99999E-07  ALPHA-  0.10000E  08  GAMMA- 


EFFECTIVENESS  TABLE 


0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0-.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.19 

0.60 

0.80 

0.90 

0.95 

0.10000E  06 
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Q 


0.10080E  08 


0. 99999E- 07  ALPHA*  0 . 1 C? 0 3 0 E  08 


EFFECTIVENESS  TABLE 


-0.  00 

0.20 

0.60 

0.80 

0.90 

-0*  00 

0.20 

0.60 

0.80 

0.90 

"0.  00 

0.20 

0.60 

0.80 

0.90 

-0.00 

0.20 

0.60 

0.80 

0.90 

-0.  00 

0.20 

0.60 

0.80 

0.90 

GAMMA* 
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EXPERIMENT  C 


0  = 


10 


-3 


q  =  10"2 

Null  at  All 

Sample 

Points 

Non  Null 
in  Upper 
Right 
Quadrant 

Not  Null 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

q  =  10"3 

Null  at  All 

Sample 

Points 

Null  at  All 

Sample 

Points 

Not  Null 

Now  Null 

for  All 
r  =  .  95 
Points 

Null  at  All 
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APPENDIX  II 


THE  BAYESIAN  EFFECTIVENESS 
OF  INFORMATION  SYSTEMS 


The  Bayesian  Value  of  an  Information  System 


The  usual  model  of  decision  making  under  uncertainty  assumes  that 
there  are  certain  states  of  nature  that  are  relevant  to  our  decision,  cer¬ 
tain  acts  that  are  open  to  us  for  choice,  and  a  utility  index  associated 
with  each  act- state  pair. 

i  k 

Let  X  denote  the  i-th  state  of  nature,  i  =  1,  .  .  .  ,  N;  A  denote  the 

k-th  act  open  to  us,  k  =  1,  .  .  ,  L;  and  uj^be  the  utility  index  assigned 

to  the  act- state  pair  (A^",  X*),  u^i  =  U(A^,  X1). 


An  information  structure  can  be  most  conveniently  characterized  as 
follows: 


X 


1 


1 1 


Y' 

q 


_M 


1M 


qNM_ 


where  Y^,  j  =  1,  .  .  .  ,  M,  is  the  j-th  message  transmitted  to  us  by  the 
information  system,  and  q—  =  P(YJ|X1)  is  the  conditional  probability 
of  the  k-th  message  given  the  fact  that  the  true  state  of  the  nature  is 
X1. 


A  rule  which  assigns  an  act  to  each  of  the  possible  messages  is 
called  a  decision  rule.  We  shall  denote  it  by  A  =  a(  Y). 

The  Bayesian  decision  rule  implies  the  following  assumptions:  (1) 
There  is  a  certain  prior  probability  associated  with  each  state  of  nature; 
we  shall  denote  it  by  P(Xx),  i  =  1,  .  .  .  ,  N.  (2)  For  each  message  observed, 
a  posterior  probability  distribution  over  the  states  of  nature  can  be  derived 
by  using  Bayes  theorem.  Let  P(X*|  Y^)  denote  the  posterior  probability  of 
X1  given  the  fact  that  Y^  has  been  observed.  Then, 
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P(X 


YJ) 


P(XX) 


P(  YJ  |  X 


*/ 1 


p(xr) 


P(  YJ 


Xr); 


(3)Let  V(A^'|  Y**)  =  P(X*|  Y^)u^.  be  the  expected  value  of  given  the 

j  * 

fact  that  Y  has  been  observed.  Then  the  Bayesian  decision  rule  says 
that  for  each  message  Y^,  one  should  select  the  act  A  =  a(Y^)  such  that 

V[a(Y‘* i)|  Yj]  =  max  V(Ak|  Yj)  . 
k 


Let 


P(YJ)  =  Jp(Xr)  •  P(YJ|xr) 
r 

be  the  probability  of  observing  the  j-th  message  given  the  prior  proba¬ 
bility  distribution  over  X  and  the  information  system  x-  Then  the 
Bayesian  value  of  x  is 


M  N 

V(x)  =  ^P(YJ)  £p(Xx|  YJ)  u[a(YJ),  X1].  (II- 1) 

j=l  i-1 


The  Bayesian  Effectiveness 


As  defined  in  the  previous  section,  the  Bayesian  decision  rule 
selects  the  act  A  =  d(Y^),  such  that 

V  [a(  Y"*)  |  Y^  ]  =  max  V(Ak|  Yj) 
k 


i.  e.  ,  such  that 


N  . 

YP(X1\  YJ)  u[a(YJ),  X1] 

i=l 
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Let  us  introduce  the  following  notation: 


where 


where 


where 


U  = 


1  1 


LI 


.  u 


IN 


LN 


uk.  =  U(Ak,  X1) 


Q  = 


lll 


*N1 


l1M 


NM 


q  =  P(  YJ|  X1) 


P  = 


11 


N1 


P(X*|  YJ) 


P(XX)  •  P(  |  X1) 

"N  ' 

Yp(Xr)  •  P(Yj|Xr) 

L _ I 

r—  1 

P(X*)  •  P(  Y^  1  X1) 

P(  yY 


1M 


NM 
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The  j-th  column  of  P,  denoted  by  [P]., 


pij 

P(XX|  Y*®) 

_PNj 

P(XN|  Y1®) 

is  the  conditional  probability  distribution  over  the  states  of  nature  if 
has  been  observed.  It  is  then  clear  that  the  j-th  column  of  UP,  denoted 
by  [UP]j,  is  the  set  of  expected  utilities  associated  with  various  acts 
conditional  on  the  occurrence  of  Y^. 

We  shall  now  define  the  operator  If  [B]  represents  a  column 
vector 


,  then  [B] 


* 


max  {b.  } . 
i 


Let  B  be  a  matrix 


Then  b"'  =  ([B]j 
B. 


[B] 


❖ 

where  fB].  denotes  the  j-th  column  of 
N  L 
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With  the  aid  of  the  operator  *:=,  we  can  define  the  Bayesian  decision 
rule  as  dlY^)  =  such  that  V(A^  I  Y^)  =  [UP]*j.  Then  the  Bayesian 
value  of  an  information  system  x  is  given  by 

M 

v(x)  =  yp(^)  [up]? 

j=l 


N 

where  PiY^)  =  ^P(Xr)  •  P(Yj|Xr). 
r=  1 


Let  E(x)  be  the  Bayesian  effectiveness  associated  with  an  informa¬ 
tion  system  X.  Then, 


E(X)  =  V(x)-V(x°) 


where  x  denotes  the  null  information  system. 

Consider  the  k-th  component  of  [UP]^.  It  is 
N  N 


Y  U(AY  X^PU1!  YJ) 
i=  1 


_ 1 _ 

P(  Y^) 


Yu(Ak,  X1)P(X1)P(Y-’|X1). 
i=l 


Let  U  =  UD 


where  D  = 


P(xj)  .  .  .  0 


N 

0  ...  P(X  ) 


is  a  diagonal  matrix  whose 


diagonal  elements  are  P(X^),  .  .  .  ,  P(X^).  Then  the  k-th  component  of 
[UP]  •  is  simply  1 /P(  Y^)  [UQ]^-;,  where  [UQlki  is  the  k-th  element  of 
UQ  and 

[UP]  =— L-  [OQ]  . 

J  P(YJ)  J 
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Since  ( 1/P(  Y*j)[UQ]  .)V  =  1/P(YJ)[UQ]  • ,  it  follows  that 

J  J 

M 

V(X)  =  ^P(Yj)[UP]J 

j=l 

M 

=  y  p(  YjH — —7 —  [uq]  .)* 

j=1  P(YJ)  3 

M 

■I"®; 

j=l 

=  (UQ)*§ 

=  [UDQ]*? 


where  §  is  a  column  vector  with  M  components  whose  values  are  all  equal 
to  1. 


Let  PQ  be  the  P  matrix  associated  with  the  null  information  system 
XQ-  Since  [UPQ]jis  the  weighted  average-with  the  weights  (PtX1)}  —  of  the 
columns  of  U  and  is  independent  of  j,  we  shall  denote  it  by  [UQ].  Then 

M 

VU  )  =  y  P(Yj)[U  ]*  =  [U  f 
o  L  o  o 

j=l 

and  E(x)  =  (UQM-[Uof  =  [UDQ]*  ?-(  UTT)* 

where  tj  is  the  vector  whose  i-th  component  is  the  prior  probability  of 
X1,  Ptx1). 
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